Optimal scheduling of virtual power plants integrating electric vehicles based on reinforcement learning

doi:10.3969/j.issn.2097-0706.2024.06.004

Abstract

Abstract:

Disorderly charging behaviors of massive electric vehicles （EVs） may cause violent fluctuations in power loads， affecting the security and stability of the power grid. With the application of vehicle to grid（V2G） technology， the scheduling method can be optimized by aggregating EV charging stations and surrounded distributed renewable energy generators into a virtual power plant（VPP）. The aggregation can improve the economy of charging behaviors and satisfaction of EV users， raise the utilization rate of distributed renewable energy， and mitigate load fluctuations in the grid. However， the overall charging or discharging load is the aggregation result of random charging or discharging behaviors of massive individual EVs， which is difficult to be accurately described by mathematical models. Thus，an interactive optimal scheduling framework based on deep reinforcement learning is presented for a VPP including EVs， with the objective of maximizing the benefit of all EV users in the VPP. The VPP control center，serving as an intelligent agent， can decide the charging and discharging of individual EVs without their detailed models. The agent continuously learns and updates its strategies through interactions with regional grids， overcoming the limitations of centralized optimal scheduling. The framework is solved by Deep Deterministic Policy Gradient（DDPG） algorithm. Simulation results show that， compared with the centralized scheduling， the proposed method improves the benefits of individual EV users， and the coordinative scheduling of EV charging/discharging loads and renewable energy outputs shaves the peak loads in the grid， and boosts the overall performance of the VPP.

Key words: virtual power plant, electric vehicle, vehicle to grid, distributed renewable energy, DDPG algorithm, optimal scheduling, reinforcement learning

CLC Number:

TK019：TM734

LI Mingyang, DOU Mengyuan. Optimal scheduling of virtual power plants integrating electric vehicles based on reinforcement learning[J]. Integrated Intelligent Energy, 2024, 46(6): 27-34.

Figures/Tables 10

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Table 1

Fig.6

Table 2

Parameters of the PSO algorithm for solving centralized optimal scheduling

参数	数值
粒子数目N	180
学习因子 $c 1, c 2$	1.5,2.0
迭代次数T	350
粒子维度D	250
速度限值 $[v m i n, v m a x]$	[-200,200]
位置限值 $[x m i n, x m a x]$	[0,350]
惯性权重w	0.99

Table 2

Table 3

Fig.7

References 25

[1]	朱磊, 黄河, 高松, 等. 计及风电消纳的电动汽车负荷优化配置研究[J]. 中国电机工程学报, 2021, 41(S1): 194-203.
	ZHU Lei, HUANG He, GAO Song, et al. Research on optimal load allocation of electric vehicle considering wind power consumption[J]. Proceedings of the CSEE, 2021, 41(S1): 194-203.
[2]	肖朝霞, 张可信, 冯冀. 含电动汽车充电站的风/光/柴独立微电网分层优化调度[J]. 天津工业大学学报, 2022, 41(4): 61-74.
	XIAO Zhaoxia, ZHANG Kexin, FENG Ji. Hierarchical optimal dispatching of wind/PV/diesel islanded microgrid with EVs charging station[J]. Journal of Tiangong University, 2022, 41(4): 61-74.
[3]	吴巨爱, 薛禹胜, 谢东亮. 电动汽车聚合商对备用服务能力的优化[J]. 电力系统自动化, 2019, 43(9): 75-81.
	WU Juai, XUE Yusheng, XIE Dongliang. Optimization of reserve service capability made by electric vehicle aggregator[J]. Automation of Electric Power Systems, 2019, 43(9): 75-81.
[4]	刘伟佳, 吴秋伟, 文福拴, 等. 电动汽车和可控负荷参与配电系统阻塞管理的市场机制[J]. 电力系统自动化, 2014, 38(24): 26-33,101.
	LIU Weijia, WU Qiuwei, WEN Fushuan, et al. A market mechanism for participation of electric vehicles and dispatchable loads in distribution system congestion management[J]. Automation of Electric Power Systems, 2014, 38(24): 26-33,101.
[5]	李红章. 基于改进粒子群算法的电动汽车充电调度优化研究[D]. 西安: 长安大学, 2021.
	LI Hongzhang. Research on optimization of electric vehicle charging scheduling based on improved particle swarm optimization algorithm[D]. Xi'an: Chang'an University, 2021.
[6]	葛晓琳, 郝广东, 夏澍, 等. 考虑规模化电动汽车与风电接入的随机解耦协同调度[J]. 电力系统自动化, 2020, 44(4): 54-62.
	GE Xiaolin, HAO Guangdong, XIA Shu, et al. Stochastic decoupling collaborative dispatch considering integration of large-scale electric vehicles and wind power[J]. Automation of Electric Power Systems, 2020, 44(4): 54-62.
[7]	黄兆元, 马海, 林俊安, 等. 含电动汽车和P2G的虚拟电厂调度优化策略[J]. 科学技术创新, 2022(6):168-171.
	HUANG Zhaoyuan, MA Hai, LIN Jun'an, et al. Scheduling optimization strategy of virtual power plant with electric vehicle and P2G[J]. Scientific and Technological Innovation, 2022(6): 168-171.
[8]	CAO Y, WANG T, KAIWARTYA O, et al. An EV charging management system concerning drivers' trip duration and mobility uncertainty[J]. IEEE Transactions on Systems Man & Cybernetics Systems, 2018, 48(4): 596-607.
[9]	袁桂丽, 王宝源. 含电动汽车的虚拟电厂经济性优化调度[J]. 太阳能学报, 2019, 40(8): 2395-2404.
	YUAN Guili, WANG Baoyuan. Economic optimal dispatch of virtual power plant with electric vehicles[J]. Acta Energiae Sloaris Sinica, 2019, 40(8): 2395-2404.
[10]	马永翔, 马少洁, 闫群民, 等. 虚拟电厂与电动汽车用户的主从博弈定价策略[J/OL]. 华北电力大学学报(自然科学版): 1-10[2024-04-18]. http://kns.cnki.net/kcms/detail/13.1212.tm.20230822.1034.006.html.
	MA Yongxiang, MA Shaojie, YAN Qunmin, et al. Master-slave game pricing strategy between virtual power plants and electric vehicle users[J/OL]. Journal of North China Electric Power University (Natural Science Edition): 1-10[2024-04-18]. http://kns.cnki.net/kcms/detail/13.1212.tm.20230822.1034.006.html.
[11]	王晛, 张华君, 张少华. 风电和电动汽车组成虚拟电厂参与电力市场的博弈模型[J]. 电力系统自动化, 2019, 43(3): 155-162.
	WANG Xian, ZHANG Huajun, ZHANG Shaohua. Game model of electricity market involving virtual power plant composed of wind power and electric vehicles[J]. Electric power system automation, 2019, 43(3): 155-162.
[12]	袁桂丽, 苏伟芳. 计及电动汽车不确定性的虚拟电厂参与AGC调频服务研究[J]. 电网技术, 2020, 44(7): 2538-2548.
	YUAN Guili, SU Weifang. Virtual power plants providing AGC FM service considering uncertainty of electric vehicles[J]. Power System Technology, 2020, 44(7): 2538-2548.
[13]	田泽禹, 沙钊旸, 赵全斌, 等. 针对温控负载变化的虚拟电厂控制策略研究[J]. 综合智慧能源, 2024, 46(1): 28-37. doi: 10.3969/j.issn.2097-0706.2024.01.004
	TIAN Zeyu, SHA Zhaoyang, ZHAO Quanbin, et al. Research on control strategy for virtual power plants in response to thermostatically controlled loads[J]. Integrated Intelligent Energy, 2024, 46(1): 28-37. doi: 10.3969/j.issn.2097-0706.2024.01.004
[14]	李茹杨, 彭慧民, 李仁刚, 等. 强化学习算法与应用综述[J]. 计算机系统应用, 2020, 29(12): 13-25.
	LI Ruyang, PENG Huimin, LI Rengang, et al. Overview on algorithms and applications for reinforcement learning[J]. Computer Systems & Applications, 2020, 29(12):13-25.
[15]	许树港. 考虑多种类型分布式能源的虚拟电厂优化调度研究[D]. 北京: 华北电力大学, 2022.
	XU Shugang. Research on optimal scheduling of virtual power plants considering multiple types of distributed energy[D]. Beijing: North China Electric Power University, 2022.
[16]	陶力, 杨夏喜, 顾金辉, 等. 基于SAC和TD3的含电动汽车虚拟电厂调度策略[J]. 电气传动, 2023, 53(9):25-34.
	TAO Li, YANG Xiaxi, GU Jinhui, et al. Scheduling strategy of virtual power plant with electric vehicle based on SAC and TD3[J]. Electric Drive, 2023, 53(9): 25-34.
[17]	卿竹雨, 安锐, 高红均, 等. 考虑分散式资源互动响应的虚拟电厂智能化调峰定价[J]. 电力自动化设备, 2023, 43(5): 96-103.
	QING Zhuyu, AN Rui, GAO Hongjun, et al. Intelligent peak regulation pricing for virtual power plant considering interactive response of distributed resources[J]. Electric Power Automation Equipment, 2023, 43(5): 96-103.
[18]	杨挺, 赵黎媛, 刘亚闯, 等. 基于深度强化学习的综合能源系统动态经济调度[J]. 电力系统自动化, 2021, 45(5): 39-47.
	YANG Ting, ZHAO Liyuan, LIU Yachuang, et al. Dynamic economic dispatch for integrated energy system based on deep reinforcement learning[J]. Automation of Electric Power Systems, 2021, 45(5): 39-47.
[19]	张子霖. 基于深度强化学习的电动汽车协调充电算法[J]. 信息技术与网络安全, 2022, 41(4): 83-89.
	ZHANG Zilin. A deep RL-based algorithm for coordinated charging of electric vehicles[J]. Information Technology and Cyber Security, 2022, 41(4): 83-89.
[20]	LILLICRAP T P, HUNT J J, PRITZEL A, et al. Continuous control with deep reinforcement learning[C]// Proceedings of 2016 International Conference on Learning Representations. ICLR, 2016: 1-14.
[21]	胡泽, 朱子晴, 卜思齐, 等. 基于深度强化学习的区域综合能源定价策略研究[J]. 综合智慧能源, 2023, 45(7): 87-96. doi: 10.3969/j.issn.2097-0706.2023.07.010
	HU Ze, ZHU Ziqing, BU Siqi, et al. Pricing strategy in district-level integrated energy market based on deep reinforcement learning[J]. Integrated Intelligent Energy, 2023, 45(7): 87-96. doi: 10.3969/j.issn.2097-0706.2023.07.010
[22]	杨维, 李歧强. 粒子群优化算法综述[J]. 中国工程科学, 2004(5): 87-94.
	YANG Wei, LI Qiqiang. Survey on particle swarm optimization algorithm[J]. Engineering Science, 2004(5): 87-94.
[23]	王鑫, 陈祖翠, 卞在平, 等. 基于粒子群优化算法的智慧微电网风光储容量优化配置[J]. 综合智慧能源, 2022, 44(6): 52-58. doi: 10.3969/j.issn.2097-0706.2022.06.006
	WANG Xin, CHEN Zucui, BIAN Zaiping, et al. Optimal allocation of a wind-PV-battery hybrid system in smart microgrid based on particle swarm optimization algorithm[J]. Integrated Intelligent Energy, 2022, 44(6): 52-58. doi: 10.3969/j.issn.2097-0706.2022.06.006
[24]	胡祖源, 靳现林, 谭雅之, 等. 基于改进粒子群算法的分布式光伏及储能系统优化配置[J]. 综合智慧能源, 2023, 45(1): 49-57. doi: 10.3969/j.issn.2097-0706.2023.01.006
	HU Zuyuan, JIN Xianlin, TAN Yazhi, et al. Optimized configuration of distributed photovoltaic and energy storage system based on improved particle swarm algorithm[J]. Integrated Intelligent Energy, 2023, 45(1): 49-57. doi: 10.3969/j.issn.2097-0706.2023.01.006
[25]	陈文颖, 刘蓓迪. 基于粒子群算法的电动汽车有序充放电优化[J]. 山东电力技术, 2023, 50(1):52-58.
	CHEN Wenying, LIU Beidi. Sequential charging and discharging optimization of electric vehicles based on particle swarm optimization[J]. Shandong Electric Power, 2023, 50(1):52-58.

时间	场景1		场景2		场景3		场景4		场景5
时间	进站EV/辆	VPP效益/元	进站EV/辆	VPP效益/元	进站EV/辆	VPP效益/元	进站EV/辆	VPP效益/元	进站EV/辆	VPP效益/元
00:00—01:00	6	39.22	6	41.22	5	39.18	7	46.96	5	34.36
01:00—02:00	12	68.70	12	76.71	14	75.96	6	46.99	10	99.57
02:00—03:00	15	69.45	9	83.43	12	71.87	11	65.95	8	70.33
03:00—04:00	11	61.30	11	74.98	10	97.34	10	81.52	11	106.01
04:00—05:00	10	66.40	6	68.75	8	53.10	8	60.63	6	59.47
05:00—06:00	8	43.33	8	60.24	7	49.83	9	60.46	8	43.84
06:00—07:00	6	32.63	7	44.32	5	43.59	7	64.63	4	30.51
07:00—08:00	6	15.17	5	45.86	7	36.87	6	41.03	6	46.26
08:00—09:00	5	10.18	7	31.83	8	25.92	6	57.86	5	19.18
09:00—10:00	6	29.36	8	29.74	7	73.73	5	35.19	8	43.96
10:00—11:00	11	45.68	5	22.29	8	58.65	12	27.76	4	32.21
11:00—12:00	7	58.75	8	40.64	14	61.13	10	23.88	10	92.72
12:00—13:00	7	66.39	9	43.69	12	82.09	7	63.27	11	101.17
13:00—14:00	9	64.68	10	85.89	10	93.44	9	86.77	9	72.97
14:00—15:00	7	45.91	8	58.67	9	40.82	7	77.28	10	-1.47
15:00—16:00	11	68.99	6	57.77	7	31.93	5	39.14	11	23.84
16:00—17:00	6	44.62	10	42.60	5	-19.07	6	54.90	11	49.37
17:00—18:00	6	47.30	8	45.34	5	49.55	8	78.28	8	53.72
18:00—19:00	9	43.74	5	23.41	0	36.68	11	56.35	11	58.56
19:00—20:00	6	36.44	9	28.03	8	22.42	7	43.52	9	25.69
20:00—21:00	4	22.08	8	34.59	6	20.68	8	4.17	5	18.60
21:00—22:00	5	26.13	8	40.46	6	27.15	5	23.25	3	23.31
22:00—23:00	3	24.90	4	20.40	4	10.61	4	9.08	2	14.55
23:00—24:00	4	33.20	3	19.16	3	14.16	6	49.33	5	34.40

场景	本文方法		集中式调度方法		本文方法效益提升/%
场景	VPP效益/元	运行时间/min	VPP效益/元	运行时间/min	本文方法效益提升/%
1	-1 064.65	7.08	-1 139.96	1.41	6.61
2	-1 120.03	8.44	-1 246.37	1.62	10.14
3	-1 097.63	7.58	-1 170.82	1.12	6.25
4	-1 198.23	7.76	-1 345.19	1.90	10.92
5	-1 153.11	8.13	-1 242.91	1.47	7.22